Presence detection and activity recognition using low-resolution passive IR sensors

Various installations and appliances used by building occupants during the day are manually operated, including office devices and kitchen appliances. Traditionally, activity monitoring systems require multi-modal sensor installations in order to monitor these types of rooms. These large multi-modal installations often require active maintenance to ensure robust operations. Due to large number of different activities performed in such rooms, traditional sensor installations often require a large amount of (different modality) sensors. This greatly increases the complexity of such systems. The work presented in this thesis introduces a framework which allows for the fine grained detection of objects and user-object interactions from their thermal signatures, using a single low resolution 2D-matrix thermopile. The use of a single sensor greatly simplifies the installation and maintenance of such sensor installations and reduces the interference perceived by the users. Privacy of the users has been taken into account during the study. As the sensor has a low resolution, typically 40 cm per pixel at 2 m distance, individual users cannot be identified. Also, due to the complexity of the output of the sensor it cannot be interpreted by people who have no knowledge of the context in which the data is obtained. As a result the privacy concerns are similar to those applicable the grid installations of traditional motion sensors. The presented framework for detecting fine grained user-object interactions consists of three main modules: 1) a sensor layer, 2) an object detection layer and 3) a classification layer. This abstraction provides a high level of flexibility in the framework. In the first module the output of the sensor is conditioned, in the second module objects are detected and tracked based on their thermal signature. In the third module the user-object interactions are classified. Furthermore, this module also classifies the current state of the objects. The state of an object is an indicator for its instantaneous energy consumption. The framework allows for the classification of 21 activities from a single sensor installment. All 21 activities involve actions commonly performed in a kitchen setting, e.g. activities involving a coffee pot or faucet. The activities are chosen based on their potential for energy savings. To evaluate the performance of the framework two data sets are recorded in the pantry area on floor 3 of the Potentiaal building at the TU/e campus. The first, scripted, data set is used to train and optimize the framework. The validation of the framework is performed on the second data set, which is a real-life scenario, consisting of 5 hours of unscripted activities. From the evaluation of the framework it is shown that the implemented algorithm is insensitive to small variations in the performed activities. It is also shown that the framework has an excellent performance of 96.4% for activities with a clear thermal signature, e.g. a coffee pot. For appliances for which the activity can only be inferred from circumstantial evidence the performance is relatively low, ranging from 11.4% for detecting opening and